fix: [no-ticket] harbor - selection/finalize integrity (idempotency, tree pooling, retries, fail-safe floor)#33
Merged
varunursekar merged 18 commits intoJul 17, 2026
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…retries, fail-safe floor) Five gaps in the champion-selection/finalize path, all observed or provoked live: 1. Idempotent finalize: the first completed result is cached and replayed on any retry. Re-running would re-rank against a DB that now contains the first finalize's own admin evals, so a retried finalize could crown a different champion than the one already reported. 2. Pooled shortlisting: recorded evals of the same commit average (not max), and commits with identical git TREES collapse into one candidate group. Max-over-rows made every re-measurement an independent lottery draw; one live optimizer farmed empty re-commits as 'clean independent lottery tickets', another refused to re-measure its champion to protect a lucky draw. Pooling makes re-measurement variance-reducing and stops identical content from stuffing the top-K shortlist. 3. Every reward-critical finalize eval (targets, shortlist re-scores, floor, baseline) retries transient failures; targets that persistently fail are floored WITH a durable target_errors marker instead of aborting finalize (a trial that ships no reward.json loses its result: happened live to an 8-hour run on a disk-full host). 4. All-error evals (score(fill_score=None) is None) retry like exceptions: an outage must never quietly become a measured 0.0. 5. Fail-safe floor: when the baseline itself cannot be measured, revert to the seed rather than shipping an unverified candidate (the floor exists to stop shipped regressions; skipping it re-opens that hole). Plus: verifier_timeout build field sizes Harbor's [verifier] timeout_sec for the whole finalize battery (the old value covered ~one eval, so Harbor could kill finalize mid-flight), and GitWorkspace.tree_hash() resolves commit content identity for the pooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…the loop (review follow-up) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shehabyasser-scale
force-pushed
the
harbor-6-selection-integrity
branch
from
July 7, 2026 13:30
f047e22 to
fdef2d2
Compare
…ity floor, trusted nested CLI) Four related fixes to keep hidden-split information and scoring authority where they belong: 1. The free baseline eval no longer rides the admin flag. engine.evaluate gains a distinct `free` parameter that waives only the budget debit; admin=True also bypassed the no_access tier gate, so the agent's one free eval could target the held-out test split and read its aggregate score off the response. The freebie is also now consumed only after a successful eval, so an infra failure no longer burns it. 2. serve.py now passes split_accesses into the EvaluationEngine. Without it the engine-side no_access gate was dormant and the budget ledger (no_access splits are unbudgeted) was the only gate, which is exactly what every unmetered path skipped. 3. k-anonymity floor on subset evals of non_viewable splits (default 5, configurable via build.yaml / serve.json). EvalSummary.mean_score over an agent-chosen singleton subset is that sample's label-derived score verbatim, so n singleton evals reconstructed a hidden split's labels wholesale. Full-split evals always pass (their aggregate is the intended surface), so splits smaller than the floor stay evaluable. The floor is advertised in status() as min_subset_samples. 4. HarborConfig.harbor_requirement: when set, the nested `harbor run` is layered over the candidate env with `uv run --with <spec>`, so the orchestrator that produces trial result.json resolves from the trusted spec, not from the candidate's own pyproject/uv.lock (one edited line there could point at a fork that fabricates results). Verified that uv's ephemeral overlay takes precedence over a conflicting project pin for both the console script and sys.path. This raises the bar, not a full boundary: agent code still imports into the nested harbor process; out-of-process verification is tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…alue, versioned re-evals, dead-attempt causes)
Four small fixes so recorded numbers say what they are:
1. summary.json now carries n_scored, n_errored, and score_se beside
mean_score: a mean over 3-of-18 scored samples, or one dominated by
errored zero-fills, is a different measurement than a clean
full-split mean, and both the agent and any auditor should see that
without per-sample access. All three are label-safe aggregates.
2. summary.json status now writes the enum VALUE ("success"), not
str(enum) ("ExperimentResultStatus.SUCCESS").
3. Result dirs are versioned per eval ({split}__{commit12}__eN instead
of wipe-and-rewrite keyed on (split, commit)): repeat measurements of
one commit (multifidelity confirms, champion re-evals) are exactly
the evidence worth comparing, and the second eval erased the first.
4. Mean-mode collation records dead_exception_types per sample: n_dead
alone hides WHY attempts died, and cause matters (rate-limit deaths
are infra noise, crashes point at the candidate; measured live,
110/129 UnicodeDecodeError deaths sat on two never-solved tasks).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fails closed on corrupt restore) Two operational fixes: 1. instruct_exhaust_budget (default True): the instruction's "unspent budget is wasted" persistence bullet becomes a build-config lever, like instruct_multifidelity. On preserves current behavior; off makes stopping-early the agent's own choice, which is the ablation arm for measuring what the exhortation itself contributes to optimizer persistence. The "scores are noisy" fact stays unconditional. 2. _load_or_build_ledger fails CLOSED on a persisted ledger that exists but cannot be parsed: metered budgets restore with zero remaining, and the unreadable file is preserved as ledger.corrupt for the operator. The old fallback restored the CONFIGURED budgets, which refunded the agent everything already spent, so any crash that corrupted the flush minted budget. A missing file is still a fresh boot; admin and finalize are unaffected either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two Greptile P2s on #30, fixed at the stack tip: - An empty transcript file no longer surfaces as "" feedback: empty candidates are skipped (an empty pane falls through to the trajectory), and if everything is empty the search moves to the next failed attempt rather than short-circuiting on "". - The no-verifier-rewards error branch (agent died before scoring) now attaches the failure transcript like any failed sample: a candidate edit that crashes the agent lands exactly here, and the transcript is the only way the optimizer can see the crash it caused. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… floor default, ordinal resume, SE naming) - Free-baseline flag is claimed BEFORE the eval await and refunded on failure: setting it only after success reopened a window where two concurrent baseline evals both resolved free (asyncio interleaves at await points). Claim-then-refund keeps both properties: concurrent callers see the claim, and a failed eval does not burn the freebie. - build_status defaults k_anonymity_floor to 5, matching the sidecar's enforcement default: a caller that forgets to pass the floor must not advertise a laxer one than gets enforced. - _route_results resumes the eval ordinal past surviving __eN dirs on a reused volume: a restarted sidecar started back at e1 and silently wiped the prior session's evidence, the exact erasure the versioned dirs exist to prevent. - score_se renamed to mean_score_se and documented: it is the SE of the zero-filled mean_score over n_samples, not of the n_scored subset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e crash vs infra outage)
Measured live in the transfer matrix: three champions scored 0/72 on an
off-model executor because their optimizers hardcoded temperature=0,
which that provider rejects. The harness handled it safely (rewards
floored, finalize shipped) but the durable record could not say WHY, and
"why" decides opposite actions: a deterministic candidate crash is a
real, reportable portability failure; an infra outage means invalidate
and re-run. Two changes:
- Collation's no-verifier-rewards error string now names the dead
attempts' exception types ("attempts died: UnsupportedParamsError
x6"). The error string is the one field that flows to the DB, the
per-sample files, and the verifier.
- _admin_eval_score returns (score, failure_cause); a floored target's
target_errors entry carries the dominant per-sample causes (frequency
summary, top 3). Diagnostics are fail-safe: any surprise result shape
degrades to a fixed string, never fails finalize.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ask crashes cluster Greptile follow-up on #37: _dominant_sample_errors keyed on the raw error string, which embeds the task name, so identical exceptions across a multi-task slice landed in 1x singletons instead of one dominant cause. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… finalize Home-model evals cannot see model-specific couplings the optimizer bakes in. Measured live in the wave-1 transfer matrix: three of five champions independently hardcoded temperature=0 (a variance trick on their home model, gpt-4.1-mini) and scored 0/72 on claude-opus-4-8, which rejects it, while looking healthy on every eval the optimization loop ever ran. The portability failure was invisible until a separate, manual, after-the-fact probe. VerificationTarget gains `model`: a target with an executor override scores the selected commit under a model it was NOT optimized on, in the same finalize battery as its home-model reward. The baseline is scored under the same override so the comparison stays like-for-like. Plumbing: build.yaml TargetSpec -> compiler -> serve.json _TargetCfg -> VerificationTarget -> engine.evaluate_admin(model=...) -> task_params ["harbor_model_override"] -> HarborRunner -m flag. The override rides task_params, so Mode A ignores it and the runner needs no new state; the shared run_constraints are copied, never mutated. Test fakes of evaluate_admin widened to accept the new kwarg (a strict signature turned the new call into a retried TypeError, flooring rewards, which is itself a nice demonstration of the floor's fail-safe). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…utage retry, key-budget alarm Three infra failure modes measured live in the E5 matrix runs, fixed at the measurement layer: - Dead attempts are classified infra vs candidate (conservative exception-type allowlist + the litellm key-budget message signature). Labels flow into dead_exception_types, error strings, and a new n_dead_infra metric. Classification never moves a score: every dead attempt still zero-fills, or faking infra would excuse failures. Exception type names are candidate-authored, so brackets are neutralized before labeling (a class named 'XError[infra]' cannot walk in pre-suffixed). - An OPT-IN, bounded, backoff-spaced within-eval retry re-measures samples whose every attempt died of a transient infra cause (the 65-second DNS blip that killed 44/72 attempts of one eval). Off by default: against an adversarial optimizer the qualifying predicate is a re-roll lever, since a stochastic candidate that raises allowlisted exceptions on failing attempts converts all-bad rounds into fresh draws. Retry rounds run in fresh sibling jobs dirs (nesting would pool dead attempts into later resumed means) and recovered samples carry an infra_retry audit marker naming the discarded attempts. - An ERROR-level alarm names key-budget exhaustion (a spent key fails every later call identically; two matrix cells of budget-exceeded zeros were nearly booked as a portability finding), with hedged wording since the signature reads candidate-process exceptions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d audit history - HarborConfig rejects infra_retry_delay_s <= 0 when retries are enabled (a zero delay silently nullified the backoff) and negative infra_retry_rounds. - The infra_retry audit marker now lists EVERY discarded round in order (discarded_rounds), not just the one immediately before recovery; recovered_round names when the sample finally measured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: [no-ticket] harbor - infra resilience (dead-attempt classification, opt-in outage retry, key-budget alarm)
feat: [no-ticket] harbor - transfer targets (per-target executor-model override at finalize)
fix: [no-ticket] harbor - floored rewards name their cause (candidate crash vs infra outage)
fix: [no-ticket] harbor - ops integrity (exhaust-budget instruction lever, fail-closed ledger restore)
fix: [no-ticket] harbor - honest measurement signals (summary qualifiers, versioned re-evals, dead-attempt causes)
fix: [no-ticket] harbor - access-tier integrity (free-baseline privilege, k-anonymity floor, trusted nested CLI)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Hardens the champion-selection/finalize path against the failure modes and exploits observed in the Pawn campaign:
GitWorkspace.tree_hash). Two live optimizers Goodharted max-over-rows selection in opposite directions: one farmed empty re-commits as "clean independent lottery tickets for max-based selection" (its words), the other refused to ever re-measure its champion to protect a lucky 0.3 draw. Pooling makes re-measurement variance-reducing, and tree-dedup stops identical content from stuffing the top-K shortlist or collecting several admin re-score draws.target_errorsmarker in the finalize payload instead of aborting finalize: a trial that ships no reward.json loses its result entirely, which happened live to an 8-hour run on a disk-full host.score(fill_score=None) is None) is an outage, indistinguishable from infrastructure failure, and is retried like an exception.verifier_timeoutbuild field: Harbor's[verifier] timeout_secmust cover the whole finalize battery (up to top-K re-scores + floor + targets + baseline attempts, each a full nested eval in Mode B), not one eval. Defaults totimeoutfor backward compatibility; docstring gives the sizing formula.Tests
tests/test_harbor_verifier.py: 34 pass, including 7 new pins: idempotent replay, same-commit pooling to mean, identical-tree dedup (no shortlist stuffing, one re-score per tree), floor fail-safe, target retry-then-floor-with-marker, transient recovery, all-error-retries. Build/serve suites green (43 pass).Stack
Based on #32 (
harbor-5-scoring-integrity). Part of the pre-paper hardening series.🤖 Generated with Claude Code
Greptile Summary
This PR hardens the Harbor champion-selection/finalize path against several failure modes observed in live campaigns, adding idempotency, commit/tree pooling, bounded retries, fail-safe floor behaviour, k-anonymity enforcement, and per-eval result versioning.
_finalize_lock+_finalize_resultcache): the first completed result is replayed verbatim on any retry, preventing re-ranking against a DB polluted by the first run's own admin evals._admin_eval_scorewith bounded retry: all reward-critical evals share one retry path; all-error evals are retried rather than silently zeroed; persistently failing targets floor rather than abort finalize.Confidence Score: 5/5
Safe to merge. All six hardening mechanisms are well-scoped, fail-closed, and backed by targeted tests pinned to live incidents.
The finalize idempotency, commit/tree pooling, retry logic, fail-safe floor, and k-anonymity gate are each independently correct and don't interact in ways that could produce a worse outcome than the pre-PR behaviour. The most sensitive path (admin/free separation and no_access gating) is explicitly fixed and tested. No correctness bugs found.
No files require special attention. The _retry_transient_infra method in runner.py and the pooling aggregation in verifier.py have minor non-correctness observations noted in comments.
Important Files Changed
Reviews (3): Last reviewed commit: "Merge pull request #34 from scaleapi/har..." | Re-trigger Greptile